fix(proxy): report eligible pool usage exhaustion as 429 by Soju06 · Pull Request #1541 · Soju06/codex-lb

Soju06 · 2026-07-30T08:38:51Z

When every account eligible for a Responses request is exhausted by an upstream usage/quota window, codex-lb previously collapsed the failure into a generic 502/503 no_accounts, which makes Codex mark persisted Goals blocked instead of usageLimited. This change classifies pool-wide exhaustion structurally (no retry-text parsing) and returns the Codex-compatible contract:

{"error": {"message": "…", "type": "usage_limit_reached", "code": "usage_limit_reached"}}

with HTTP 429, plus error.resets_at only when account selection holds an authoritative upstream reset for an actually exhausted window (never the capped human retry hint or a synthesized fallback).

Closes #1246.

Supersedes #1247 — original implementation by @glopyglerky, taken over per the 07-27 maintainer notice. The author's 10 commits are preserved with original authorship; takeover commits sit on top.

Behavior matrix (#1246)

one exhausted account + one usable account → internal failover, no quota error
every eligible account usage-limited → 429 usage_limit_reached (+ authoritative resets_at)
all accounts paused/deactivated/reauth-required (no usage evidence) → existing no_accounts semantics
local response-create/stream caps (and every LOCAL_OVERLOAD_CODES member, including codes added later such as api_key_stream_fair_share from feat(proxy): congestion-aware per-API-key fair-share stream admission #1536) → existing 429 rate_limit_error
previous-response owner exhausted, safe failover possible → internal retry/failover
owner unavailable while the wider pool has capacity → existing continuity 502
owner and entire wider pool exhausted → 429 usage_limit_reached
usage-limit selection failures are terminal on streaming/bridge/WebSocket paths — no capacity-recovery waits

Rebase notes

Rebased onto current main across the _load_balancer/ package extraction, fix(proxy): spill unanchored forks on account caps #1499 (unanchored-fork spillover) and fix(proxy): release idle bridge sessions' account stream leases #1476 (idle bridge lease release); the author's balancer-loop changes were ported into unbound_selection.py/sticky_selection.py.
Verified against fix(proxy): sequence synthetic Responses failures #1479 failure-envelope sequencing: the pre-stream 429 exits as an HTTP response before _normalize_public_responses_stream; streamed selection failures keep the sequenced synthetic response.created → response.failed contract (pinned by integration test).
selection_failure_response resolves local capacity codes against the canonical LOCAL_OVERLOAD_CODES registry so the new 429 mapping is strictly for upstream usage exhaustion.

OpenSpec

openspec/changes/report-pool-usage-exhaustion/ (delta on responses-api-compat), extended for authoritative-reset-only resets_at, terminal usage-limit failures, local-capacity contract preservation, and owner-scope continuity. openspec validate report-pool-usage-exhaustion --strict passes.

Testing

ruff check + ruff format --check + scripts/check_proxy_architecture.py + uv run ty check: pass
Targeted suites: tests/unit/test_load_balancer*.py, test_selection_errors.py, test_openai_errors.py, test_proxy_http_bridge.py, test_proxy_utils.py, tests/integration/test_proxy_responses.py, test_http_responses_bridge.py, test_proxy_api_extended.py — 1,919 tests passed
New externally-routed regressions assert the full 503-vs-429 matrix at /v1/responses and /backend-api/codex/responses, resets_at presence/omission, and the fix(proxy): sequence synthetic Responses failures #1479 sequenced SSE failure contract.

🤖 Generated with Claude Code

Soju06 · 2026-07-30T08:39:07Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d73fd3869d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Komzpa · 2026-07-31T22:14:23Z

@codex review

chatgpt-codex-connector · 2026-07-31T22:14:35Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

jaekwonhong · 2026-08-05T07:51:01Z

@codex review

chatgpt-codex-connector · 2026-08-05T07:59:20Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 9eb44abb93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The takeover of #1247 hardens selection_failure_response: instead of a private duplicate of the account-cap code set, local capacity codes are resolved via app.core.resilience.overload.LOCAL_OVERLOAD_CODES. This keeps every local overload code (including codes added later, such as the congestion fair-share code from #1536) on the stable 429 rate_limit_error contract, and guarantees the new usage_limit_reached 429 mapping is applied strictly to upstream usage/quota exhaustion of the whole eligible pool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Adds externally-routed regressions for the takeover of #1247: - /v1/responses and /backend-api/codex/responses return HTTP 429 with error.type = error.code = usage_limit_reached (and authoritative error.resets_at) when every eligible account is usage-exhausted - resets_at is omitted when selection has no authoritative reset - paused/deactivated/reauth-only pools keep the pre-existing no_accounts semantics, and the synthetic SSE failure keeps the #1479 sequenced response.created/response.failed SDK stream contract - one usable account still fails over with no error Also tightens the OpenSpec delta: authoritative-reset-only resets_at, terminal (non-waitable) usage-limit failures, local capacity codes keep their rate_limit_error contract, owner-scoped exhaustion keeps continuity semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Soju06 mentioned this pull request Jul 30, 2026

fix(proxy): report eligible pool usage exhaustion as 429 #1247

Open

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread app/modules/proxy/_load_balancer/sticky_selection.py

Comment thread app/modules/proxy/_service/streaming/retry.py

Comment thread app/core/balancer/logic.py Outdated

Komzpa added 🤖 codex: needs work [@codex review] raised an issue and removed 🤖 codex: needs work [@codex review] raised an issue labels Jul 30, 2026

Komzpa added 🤖 codex: needs work [@codex review] raised an issue and removed 🤖 codex: needs work [@codex review] raised an issue labels Jul 31, 2026

Soju06 mentioned this pull request Aug 4, 2026

Penalize eventless HTTP bridge accounts #1574

Closed

Soju06 force-pushed the fix/pool-usage-exhaustion-429 branch from acbf7a4 to 9eb44ab Compare August 4, 2026 07:41

Soju06 mentioned this pull request Aug 6, 2026

fix(proxy): allow fresh burn-first usage drain #1603

Closed

Soju06 force-pushed the fix/pool-usage-exhaustion-429 branch 2 times, most recently from cd1f47a to 97194be Compare August 6, 2026 05:46

Omnidia Agent and others added 14 commits August 6, 2026 08:12

fix(proxy): preserve pool usage-limit semantics

bab8fac

style: normalize pool exhaustion imports

3984c99

fix(proxy): preserve structured retry hints

2f8bb5c

docs(openspec): cover pool usage exhaustion errors

8d3dd7a

fix(ci): satisfy pool usage PR gates

0148241

fix(balancer): tighten usage-limit exhaustion reporting

81c4723

fix(balancer): preserve usage-limit reset metadata

3ce1c9c

fix(balancer): require usage evidence for pool limits

07bc842

fix(balancer): preserve cap semantics for usage limits

8375597

fix(balancer): preserve pressure usage priorities

479d83a

style: format unbound selection cap fallback

f86c58c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(proxy): honor usage exhaustion boundaries

cdf2a27

Soju06 force-pushed the fix/pool-usage-exhaustion-429 branch from 97194be to cdf2a27 Compare August 6, 2026 08:17

Soju06 merged commit 897ae15 into main Aug 6, 2026
36 checks passed

This was referenced Aug 6, 2026

chore: release v1.23.0-beta.4 #1624

Merged

chore(main): release 1.23.0 #1472

Open

Navigation Menu

fix(proxy): report eligible pool usage exhaustion as 429 - #1541

fix(proxy): report eligible pool usage exhaustion as 429#1541
Soju06 merged 14 commits into
mainfrom
fix/pool-usage-exhaustion-429

Soju06 commented Jul 30, 2026

Uh oh!

Soju06 commented Jul 30, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Komzpa commented Jul 31, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 31, 2026

Uh oh!

jaekwonhong commented Aug 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Soju06 commented Jul 30, 2026

Behavior matrix (#1246)

Rebase notes

OpenSpec

Testing

Uh oh!

Soju06 commented Jul 30, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Komzpa commented Jul 31, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 31, 2026

Uh oh!

jaekwonhong commented Aug 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants